Efficient Massively Parallel Transport Sweeps

The full-domain “sweep,” in which all angular fluxes in a problem are calculated given previous-iterate values only for the volumetric source, forms the foundation for many iterative methods that have desirable properties [1]. One important property is that iteration counts do not grow with mesh refinement [1]. The sweep solution on parallel machines is complicated by the dependency of a given cell on its upstream neighbors. A simple task dependence graph (TDG) for a single quadrature direction in a 2D example (Fig.1) illustrates the issue: tasks at a given level of the graph cannot be executed until some tasks finish on the previous level. The KBA algorithm [2] partitions the problem by assigning a column of cells to each processor, indicated by the four diagonal task groupings in Fig.1. KBA parallelizes over planes perpendicular to the sweep direction—over the breadth of the TDG. Early and late in a single-direction sweep, some processors are idle, as in stages 1-3 and 9-11 in Fig.1. In this example, parallel efficiency for an isolated singledirection sweep could be no better than 8/11 = 0.73. KBA is much better, because when a processor finishes its tasks for the first direction it begins its tasks for the next direction in the octant-pair with the same sweep ordering. That is, each processor begins a new TDG as soon as it completes its work on the previous TDG, until all directions in the octant-pair finish. This effectively lengthens the “pipe” and increases efficiency. If there were 2M directions in the octant pair, then the pipe length is 2M × 8 in this example, and the efficiency could be up to (2M × 8)/(3 + 2M × 8). KBA’s pipe-fill penalty grows as processor count grows, even if cell count grows proportionally. The width of the TDG grows only as P , so traditional KBA eventually runs out of parallelism to exploit. These issues fuel the common belief that sweeps cannot perform well in parallel beyond a few thousand processing elements. One purpose of this summary is to help dispel this belief. A sweep algorithm is defined by its partitioning (dividing the domain among processors), aggregation (grouping cells, directions, and energy groups into “tasks”), and scheduling (choosing which task to execute if more than one is available). The work presented here follows that